Gaze and saccade based graphical manipulation

ABSTRACT

According to the invention, a system for presenting graphics on a display device is disclosed. The system may include an eye tracking device and a graphics processing device. The eye tracking device may be for determining a gaze point of a user on a display device. The graphics processing device may be for causing graphics to be displayed on the display device. The graphics displayed on the display device may be modified such that the graphics are of higher quality in an area including the gaze point of the user, than outside the area.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/270,783, filed Sep. 20, 2016, which is a continuation-in-part of U.S.patent application Ser. No. 14/197,171, filed Mar. 4, 2014, now U.S.Pat. No. 9,665,171, which claims priority to U.S. Provisional PatentApplication No. 61/772,366, filed Mar. 4, 2013, the entire disclosuresof which are hereby incorporated by reference, for all purposes, as iffully set forth herein.

BACKGROUND OF THE INVENTION

Graphical items may be used on a display to show data and information toa viewer. These graphical items may include text, images, and video.Graphical items in the area of computing are well known and have been inuse for many years. Recently, showing three dimensional (3D) graphicalitems on a display has increased in importance in areas such as gaming,modeling and movies.

When displaying graphical items, a system such as a computer uses aprocessor in combination with memory to display the items on a screen orother display device. Methods for displaying graphical items vary, buttypically they rely on a computer interpreting commands to control agraphics processing device that provides graphical items for display.The graphics processing device typically contains custom hardware forthis purpose including a processor and memory. In some computer systemsthe graphics processing device is fully integrated, and in others it isprovided as a separate component known as a graphics card.

Graphics processing devices have limits on their processing power,usually quantified in terms of the amount of graphical items that can bedisplayed on a screen at any given time. This is typically limited bythe capabilities of the hardware embodying the graphics processingdevice, including processors, memory, and communication channels.Additionally, the amount of graphical items able to be displayed on ascreen at a given point can be limited by communication limits betweenthe graphics processing device and computer.

In many scenarios that require graphical items be displayed on a screen,a user only focuses on a portion of the screen, and therefore only aportion of the graphical items, an any given time. Meanwhile, othergraphical items continue to be displayed on the remaining portions ofthe screen, which the user is not focused on. This wastes valuablegraphics processing device resources to produce graphical items thatcannot be fully appreciated by the user because the visual acuity of ahuman drops dramatically outside those images immediately focused on.

BRIEF DESCRIPTION OF THE INVENTION

In some embodiments, a system for presenting graphics on a displaydevice is provided. The system may include an eye tracking device and agraphics processing device. The eye tracking device may be fordetermining a gaze point of a user on a display device. The graphicsprocessing device may be for causing graphics to be displayed on thedisplay device. The graphics displayed on the display device may bemodified such that the graphics are of higher quality in an areaincluding the gaze point of the user, than outside the area.

In another embodiment, a method for presenting graphics on a displaydevice is provided. The method may include determining, with an eyetracking device, a gaze point of a user on a display device. The methodmay also include causing, with a graphics processing device, graphics tobe displayed on the display device. The graphics displayed on thedisplay device may be modified such that the graphics are of higherquality in an area including the gaze point of the user, than outsidethe area.

In another embodiment, a non-transitory machine readable medium havinginstructions thereon for presenting graphics on a display device isprovided. The instructions may be executable by one or more processorsto perform a method. The method may include determining a gaze point ofa user on a display device, and causing graphics to be displayed on thedisplay device. The graphics displayed on the display device may bemodified such that the graphics are of higher quality in an areaincluding the gaze point of the user, than outside the area.

In another embodiment, a system for presenting graphics on a displaydevice is provided. The system may include an eye tracking device in auser wearable device for determining a gaze point of a user on a displaydevice

a processing device for causing graphics to be displayed on the displaydevice;

wherein the graphics displayed on the display device are modified suchthat the graphics are of higher quality in an area including the gazepoint of the user, than outside the area.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in conjunction with the appendedfigures:

FIG. 1 is a block diagram of one possible system of the invention formodifying an image based on a user's gaze point;

FIG. 2 is a view of a display device of the invention in which imagemodification is occurring in response to a user's gaze point;

FIG. 3A is a diagram of how image quality may continuously vary within amodified image area;

FIG. 3B is a diagram of how image quality may vary in steps within amodified image area;

FIG. 4 is a view of a display device of the invention in which imagemodification is occurring in response to a detected change in a user'sgaze point;

FIG. 5 is flow diagram of one possible method of the invention formodifying an image based on a user's gaze point;

FIG. 6 is a block diagram of an exemplary computer system capable ofbeing used in at least some portion of the devices or systems of thepresent invention, or implementing at least some portion of the methodsof the present invention;

FIGS. 7A and 7B are diagrams showing a change in the location of an areaon a display device according to the change in a user's gaze point; and

FIG. 8 is a diagram demonstrating an eccentricity angle.

In the appended figures, similar components and/or features may have thesame numerical reference label. Further, various components of the sametype may be distinguished by following the reference label by a letterthat distinguishes among the similar components and/or features. If onlythe first numerical reference label is used in the specification, thedescription is applicable to any one of the similar components and/orfeatures having the same first numerical reference label irrespective ofthe letter suffix.

DETAILED DESCRIPTION OF THE INVENTION

The ensuing description provides exemplary embodiments only, and is notintended to limit the scope, applicability or configuration of thedisclosure. Rather, the ensuing description of the exemplary embodimentswill provide those skilled in the art with an enabling description forimplementing one or more exemplary embodiments. It being understood thatvarious changes may be made in the function and arrangement of elementswithout departing from the spirit and scope of the invention as setforth in the appended claims.

Specific details are given in the following description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits,systems, networks, processes, and other elements in the invention may beshown as components in block diagram form in order not to obscure theembodiments in unnecessary detail. In other instances, well-knowncircuits, processes, algorithms, structures, and techniques may be shownwithout unnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that individual embodiments may be described as aprocess which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay describe the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations may be re-arranged. A process may beterminated when its operations are completed, but could have additionalsteps not discussed or included in a figure. Furthermore, not alloperations in any particularly described process may occur in allembodiments. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination corresponds to a return of the functionto the calling function or the main function. Any detail present in onediscussed embodiment may or may not be present in other versions of thatembodiment or other embodiments discussed herein.

The term “machine-readable medium” includes, but is not limited toportable or fixed storage devices, optical storage devices, wirelesschannels and various other mediums capable of storing, containing orcarrying instruction(s) and/or data. A code segment ormachine-executable instructions may represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc. may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

Furthermore, embodiments of the invention may be implemented, at leastin part, either manually or automatically. Manual or automaticimplementations may be executed, or at least assisted, through the useof machines, hardware, software, firmware, middleware, microcode,hardware description languages, or any combination thereof. Whenimplemented in software, firmware, middleware or microcode, the programcode or code segments to perform the necessary tasks may be stored in amachine readable medium. A processor(s) may perform the necessary tasks.

In some embodiments of the invention, and with reference to FIG. 1, asystem 100 for presenting graphics on a display device 110 is provided.System 100 may include an eye tracking device 120 and a graphicsprocessing device 130. In some embodiments, the system may also includea processor/computer 140 which communicates with, and controls, graphicsprocessing device 130. In some embodiments, any function of graphicsprocessing device 130 may be performed, in whole or in part, byprocessor/computer 140. Merely by way of example, eye tracking device120 may be provided integral to, or in addition to, a personal computer140 having graphics processing device 130 and a central processing unit(in some configurations, graphics processing device 130 and the centralprocessing unit are integrated). In other embodiments, eye trackingdevice 120 may be provided integral to, or in addition to, a gamingconsole 140 or other device having graphics processing device 130 and acentral processing unit. Examples of gaming consoles include thoseproduced and available from Microsoft™, Nintendo™, or Sony™. In otherembodiments, the eye tracking device 120 may be provided integral to, orin addition to, a wearable headset such as a Virtual Reality (VR) orAugmented Reality (AR) or the like. Examples of wearable headsetsinclude those produced and available under the names Oculus Rift™, HTCVive™, Sony PlaystationVR™ and Fove™. Thus, embodiments of the inventionmay be applied to the presentation of graphics in any number of possibledevices and applications, including video display, video games, videoproduction and editing, video communications, computer aided draftingand design, etc.

Eye tracking device 120 may be for determining at least one of a gazepoint of a user on display device 110, or a change in the gaze point ofthe user on display device 110. Eye tracking devices and methods,sometimes referred to as gaze detection systems and methods, include,for example, products produced and available from Tobii Technology AB,and which operate by using infrared illumination and an image sensor todetect reflection from the eye of a user. An example of such a gazedetection system is described in U.S. Pat. No. 7,572,008, which ishereby incorporated by reference, for all purposes, as if fully setforth herein. Other alternative gaze detection systems may also beemployed by the invention, regardless of the technology behind the gazedetection system. Eye tracking device 120 may employ its own processoror the processor of another device (i.e., processor/computer 140) tointerpret and process data received. When an eye tracking device isreferred to herein, both possible methods of processing data arereferred to.

Graphics processing device 130 employed by various embodiments of theinvention may be for causing an image to be displayed on display device110. Graphics processing device 130 may modify what image is displayedon display device 110 based at least in part on the gaze point of theuser on display device 110, or a change in the gaze point of the user ondisplay device 110, as determined by eye tracking device 120. While insome embodiments a separate non-included or non-integrated displaydevice will be controlled by the graphics processing device 130, otherembodiments may include or integrate display device 110.

The way in which the image displayed on display device 110 may bemodified by graphics processing device 130 may vary depending on theembodiment, but regardless, the way in which the image is displayed maybe intended to increase the image quality of portions of the image onwhich a user's gaze, or focused gaze, is directed, relative to thoseportions of the image to which the user's gaze, or focused gaze, is notdirected. In this manner, the use of available resources of graphicsprocessing device 130, and/or other system resources, are maximized todeliver image quality where it matters most on display device 110. Todemonstrate, FIG. 2 illustrates a display device 110 showing a user'sgaze point 210 and an area 220 around user's gaze point 210 in whichembodiments of the invention may increase the quality of the imagerelative to the remaining area 230 of the display device 110. Thus, invarious embodiments of the invention, the quality of the image producedacross display device 110 may be increased in area 220 relative toremaining area 230.

When “modification” of an image presented on display device 110 isdiscussed herein, it shall be understood that what is intended is that asubsequent image displayed on display device 110, is different than aprior image displayed on display device 110. Thus, graphics processingdevice 130 and display device 110, or other device(s) discussed herein,“modify” an image by causing a first image to be displayed and then asecond image to be displayed which is different than the first image.Any other change of an image discussed herein, for example, increasingor decreasing of image quality, shall also be understood to mean that asubsequent image is different than a prior image. Note that a change ormodification of an image may include changing or modifying only aportion of the image. Thus, some portions of a prior image may be thesame as a subsequent image, while other portions may be different. Inother situations, the entirety of a prior image may be different than asubsequent image. It shall be understood that the modification of anarea or an entirety of an image does not necessarily mean every finiteportion of the area or entirety are changed (for example, each pixel),but rather that the area or entirety may be changed in some potentiallyconsistent, predefined, or ordered manner (for example, the quality ofthe image is changed).

Increasing the quality of the image may include increasing the qualityof any one or more of the below non-exclusive list of graphicalcharacteristics, in addition to other possible characteristics known inthe art:

-   -   Resolution: The number of distinct pixels that may be displayed        in one or more dimensions. For example, “1024×768” means 1024        pixels displayed in height and 768 pixels displayed in width.    -   Shading: Variation of the color and brightness of graphical        objects dependent on the artificial lighting projected by light        sources emulated by graphics processing device 130.    -   Texture-mapping: The mapping of graphical images or “textures”        onto graphical objects to provide the objects with a particular        look. The resolution of the textures influence the quality of        the graphical object to which they are applied.    -   Bump-mapping: Simulation of small-scale bumps and rough        gradients on surfaces of graphical objects.    -   Fogging/participating medium: The dimming of light when passing        through non-clear atmosphere or air.    -   Shadows: Emulation of obstruction of light.    -   Soft shadows: Variance in shadowing and darkness caused by        partially obscured light sources.    -   Reflection: Representations of mirror-like or high gloss        reflective surfaces.    -   Transparency/opacity (optical or graphic): Sharp transmission of        light through solid objects.    -   Translucency: Highly scattered transmission of light through        solid objects.    -   Refraction: Bending of light associated with transparency.    -   Diffraction: Bending, spreading and interference of light        passing by an object or aperture that disrupts the light ray.    -   Indirect illumination: Surfaces illuminated by light reflected        off other surfaces, rather than directly from a light source        (also known as global illumination).    -   Caustics (a form of indirect illumination): Reflection of light        off a shiny object, or focusing of light through a transparent        object, to produce bright highlights on another object.    -   Anti-aliasing: The process of blending the edge of a displayed        object to reduce the appearance of sharpness or jagged lines.        Typically an algorithm is used that samples colors around the        edge of the displayed object in to blend the edge to its        surroundings.    -   Frame rate: For an animated image, the number of individual        frames presented during a certain period of time to render        movement within the image.    -   3D: Visual and temporal characteristics of an image which cause        the image to appear to be three dimensional to a viewer.    -   Animation quality: When an animated image is presented, the        detail of the animated image is decreased in the periphery of        view.    -   Post processing effects quality    -   Refraction quality    -   Three dimensional object quality: Polygons shown based on the        distance of the object to the virtual camera

The size and shape of the area of the image which may be modified toappear in greater quality can vary depending on the embodiment. Merelyby way of example, the shape of the area may be circular, oval, square,rectangular, or polygonal. In some embodiments, the quality of the imagewithin the area may be uniformly increased. In other embodiments, theincrease in quality of the image may be greatest at the center of thearea (i.e., proximate to the gaze point), and decrease towards the edgesof the area (i.e., distal to the gaze point), perhaps to match thequality of the image surrounding the area. To demonstrate, FIG. 3A showshow image quality may decrease in a linear or non-liner continuousmanner from the center of a gaze area outward, while FIG. 3B shows howimage quality may decrease in a stepped manner from the center of a gazearea outward.

In some embodiments, modifying the image displayed on display device 110may occur in response to the detection of a change in the gaze point.This may occur in a number of fashions, some of which are describedbelow.

In some embodiments, an entirety of the image may be modified during theperiod of change in the gaze point of the user, and once the change inthe gaze point of the user ceases, either the area around end gaze pointof the user or a remainder of the image (portions of the image notaround the end gaze point) may be modified. Merely by way of example, inone embodiment, the quality of the entire image may be increased duringmovement of the user's gaze (sometimes referred to as a saccade), butthe increase in quality may only be sustained in an area around theuser's end gaze point once the saccade is complete (i.e., the quality ofthe remainder of the image may be decreased upon completion of thesaccade). In a different embodiment, the quality of the entire image maybe decreased during a saccade, but the decrease in quality may only besustained areas besides around the user's end gaze point once thesaccade is complete (i.e., the quality of the area of the image aroundthe user's end gaze point may be increased upon completion of thesaccade).

Additionally, the use of other system resources, including for exampleprocessor/computer 140 and related resources, may also be modifiedduring a user's saccade. For example, non-graphical operations may besupplemented by the resources of processor/computer 140 and graphicsprocessing device 130, during a saccade. More specifically, during asaccade, non-graphical calculations necessary for other systemoperations may proceed at greater speed or efficiency because additionalresources associated with processor/computer 140 and graphics processingdevice 130 are made available for such operations.

In some embodiments, modifying the image displayed on display device 110may include modifying a portion of the image in an area around ananticipated gaze point of the user, potentially by increasing thequality thereof. The anticipated gaze point may be determined based onthe change in the gaze point of the user. To determine the anticipatedgaze point of a user, eye tracking device 120 and/or another processor(i.e., the computer or game consoler's processor), may determine a rateof the change in the gaze point of the user on display device 110, anddetermine the anticipated gaze point based at least in part on this rateof the change.

The rate of change of the gaze point of the user, also referred to asthe velocity or speed of a saccade by the user is directly dependent onthe total change in the gaze point of the user (often referred to as theamplitude of the saccade). Thus, as the intended amplitude of a user'ssaccade increases, so does the speed of the saccade. While the saccadeof a human user can be as fast as 900°/second in humans, for saccades ofless than or about 60°, the velocity of a saccade is generally linearlyand directly dependent on the amplitude of the saccade. For example, a10° amplitude is associated with a velocity of 300°/second and a 30°amplitude is associated with a velocity of 500°/second. For saccades ofgreater than 60°, the peak velocity starts to plateau toward the maximumvelocity attainable by the eye (900°/second). In response to anunexpected stimulus, a saccade normally takes about 200 milliseconds(ms) to be initiated and then lasts from about 20 to about 200 ms. Basedon these relationships between saccade speed and amplitude, embodimentsof the invention may determine anticipated gaze points based on saccadevelocity. Other predetermined models of mathematical relationshipsbetween saccade speed and amplitude may also be employed by variousembodiments of the invention to determine an anticipated gaze point.

In some embodiments, the portion of the image modified around theanticipated gaze point may also include the portion of the image aroundthe original gaze point (i.e., the gaze point from which the user'ssaccade started). While the shape of the portion of the image modifiedmay be any of those shapes described above, in some embodiments it maybe a triangle or a trapezoidal shape having a progressively greaterwidth perpendicular to a direction of the saccade as shown in FIG. 4.

In FIG. 4, display device 110 is shown, and an initial user gaze point410 is shown thereon. Prior to any change in initial gaze point 410,embodiments of the invention may provide increased graphics quality inarea 420. When a user saccade, represented by arrow 430, is detected byeye tracking device 120, the size and shape of area 420 may change toaccommodate both initial gaze point 410 and anticipated gaze point 440.The changed area 450, while being triangular and/or trapezoidal in thisembodiment, may be shaped and sized differently in other embodiments.Merely by way of example, an entire side of display device 110 from theinitial gaze point to the edges of the display in the direction of thesaccade may also be included in changed area 450 to account for morepossibilities of where the user's gaze point may end. In otherembodiments, a circular, oval, or square changed area 450 may beprovided. In yet other embodiments, changed area 450 may includeseparate and distinct areas around the initial gaze point 410 andanticipated gaze point 440.

In some embodiments, the size or shape of the area around the gaze pointfor which an image is modified (or which remains unmodified from aheightened quality in various embodiments), is dynamic. This may occurbased at least in part on any number of factors, including the currentlocation of the gaze point relative to the image or display device.Merely by way of example, if a user moves their gaze point to a certainportion of the screen, a predefined portion of the screen may bemodified via increased quality therein (for example, a corner portion ofthe display having a map of a virtual area in a video game). In someembodiments, if enough user saccades having one or more predefinedcharacteristics are detected in predefined amount of time, the entiretyof the display may be modified to be rendered in greater quality.

The performance of a computing system may be directly influenced by theconsumption of resources the system has at its disposal. These resourcesinclude, but are not limited to, processing power, memory size, memoryaccess speed, and computer bus speed. The display of information such asimages and other graphical items may directly require the use of suchresources. The higher the quality of this information, as has beenpreviously described, the greater the amount of resources required, orthe greater level of strain on existing resources. The present inventionseeks to decrease the consumption of these resources by allocatinggraphical processing and execution resources first and primarily toareas of display device 110 that can be readily perceived in highdefinition by a user, as the areas in which high definition informationis actually displayed. Other areas of the display device, which will notbe, or cannot easily be, perceived in high definition by a user may beallocated a lesser or remaining amount of resources. Due to latencybetween the output of information to display device 110 and the speed atwhich a user can move their eyes and perceive information, it may bedesirable to provide a system in which the user does not perceive thatthere is any change to the quality or definition of information beingdisplayed on the display device 110.

In some embodiments, the gaze point information may be determined by, orbased on information from, eye tracking device 120, and may be filteredto minimize the change in areas around the gaze point for which an imagemay be modified. Referring to FIG. 7A the embodiment includes displaydevice 110 comprising an area 800 containing a sub-area 810. Anobjective of this and other embodiments may be to maintain highdefinition and/or other improved graphical rendering qualities insub-area 810 and/or area 800 around any determined gaze points 210. Animage may thus be modified in area 800 such that it contains greaterquality graphics or other modified parameters as previously described.If it is determined that a gaze point 210 remains within sub-area 810,the quality of the graphics or the like in area 800 may be modified suchthat the graphical quality of the images in area 800 are displayed at ahigher quality or the like than images outside of area 800 on displaydevice 110. If it is determined that the gaze point is located outsideof sub-area 810, as shown in FIG. 7B, a new area 800A is definedcontaining a new sub-area 810A. New area 800A may then be modified tocontain higher quality graphics or other parameters. The invention thenrepeats, in that if it is determined that the gaze point remains withinnew sub-area 810A, area 800A remains constant, however if the gaze pointis detected outside of new sub-area 810A, area 800A is redefined.

In these or other embodiments, filtering of gaze information mayperformed to ensure that relocation of area 800 is necessary. Forexample, the system may determine a gaze point 210 is located outsidethe sub-area 810 however it may perform no action (such as relocatingthe area 800) until a predetermined number of gaze points 210 arelocated outside the sub-area (for example 2, 5, 10, 50). Alternatively,the system could temporarily enlarge area 800 until it is certain thegaze point 210 is located within a certain area. Additionally,predefined time periods may be established to determine if gaze points210 have moved outside of sub-area 810 for at least those time periodsprior to enlarging or changing are 800.

In some embodiments, the size of area 810 may be based on aneccentricity angle of about 1-5 degrees from the currently determinedgaze point. In exemplary embodiments, the angle may be about 3 degrees.Although the concept of an eccentricity angle would be well understoodby a person of skill in the art, for demonstration purposes, its use inthe present embodiment will now be described with reference to FIG. 8.

The eccentricity angle θ represents a person's fovea vision. In someembodiments, it may be preferable that area 800 is larger than the arealabeled “object” in FIG. 8. This means that there may be a highprobability that while a person's gaze point 210 remains within thesub-area 810, that person will not be able to perceive informationoutside the area 800 at high quality. The size of the area labeled“object” on display device 110 is primarily dependent on the physicalsize of display device 110 and the distance between the eye(s) anddisplay device 110.

The size of area 800 may be modified to affect performance, as thegreater the size of area 800, the more system resources are required torender graphics at higher quality. By way of example, the area 800 maybe of such size so as to fit within the display 2 to 12 times. This sizemay be optionally linked directly to the size of display device 110, orto the eccentricity angle, such that the size may scale efficiently. Ina further embodiment, gaze point 210 may be determined after adjustinginformation obtained by eye tracking device 120 to remove noise.Different individuals may have different levels of noise in the gazeinformation obtained by eye tracking device 120 (for example, due towandering of their eye(s) about the gaze point). If an individual has alow level of noise, area 800 may be smaller, and thus performance of thesystem on which the present embodiment is being practiced may beincreased.

In another embodiment of the invention, a non-transitory computerreadable medium having instructions thereon for presenting graphics ondisplay device 110 is provided. The instructions may be executable byone or more processors to at least display an image on display device110. The instructions may also be executable to receive information fromeye tracking device 120 indicative of at least one of a gaze point of auser on display device 110, or a change in the gaze point of the user ondisplay device 110. The instructions may further be executable to causegraphics processing device 130 to modify the image displayed on displaydevice 110 based at least in part on the gaze point of the user ondisplay device 110, or the change in the gaze point of the user ondisplay device 110. Thus, a non-transitory computer readable medium ableto implement any of the features described herein in relation to otherembodiments is also provided.

In another embodiment of the invention, a method 500 for presentinggraphics on display device 110 is provided as shown in FIG. 5. At step510, method 500 may include displaying an image on display device 110.At step 520, method 500 may also include receiving information from eyetracking device 120 indicative of at least one of a gaze point of a useron display device 110, or a change in the gaze point of the user ondisplay device 110. At step 530, method 500 may further include causinggraphics processing device 130 to modify the image displayed on displaydevice 110 based at least in part on the gaze point of the user ondisplay device 110, or the change in the gaze point of the user ondisplay device 110. Step 530 may include, at step 533, increasing thequality of the image in an area around the gaze point of the user,relative to outside the area. Step 530 may also include, at step 536,decreasing the quality of the image outside an area around the gazepoint of the user, relative to inside the area. Thus, a method toimplement any of the features described herein in relation to otherembodiments is also provided.

In some embodiments, the systems and methods described herein may betoggled on and off by a user, possibly to account for multipleadditional viewers of display device 110 being present. In otherembodiments, the systems and methods described herein may automaticallytoggle on when only one user is viewing display device 110 (as detectedby eye tracking device 120), and off when more than one user is viewingdisplay device 110 (as detected by eye tracking device 120).Additionally, in some embodiments, the systems and methods describedherein may allow for reduction in rendering quality of an entire displaydevice 110 when no viewers are detected, thereby saving system resourcesand power consumption when display device 110 is not the primary focusof any viewer.

In other embodiments, the systems and methods described herein may allowfor modifying multiple portions of an image on display device 110 toaccount for multiple viewers as detected by eye tracking device 120. Forexample, if two different users are focused on different portions ofdisplay device 110, the two different areas of the image focused on maybe rendered in higher quality to provide enhanced viewing for eachviewer.

In yet other embodiments, data associated with an image may inform thesystems and methods described herein to allow prediction of which areasof an image may likely be focused on next by the user. This data maysupplement data provided by eye tracking device 120 to allow for quickerand more fluid adjustment of the quality of the image in areas likely tobe focused on by a user. For example, during viewing of a sportingevent, a picture-in-picture of an interview with a coach or player maybe presented in a corner of the image. Metadata associated with theimage feed may inform the systems and methods described herein of thelikely importance, and hence viewer interest and likely focus, in thesub-portion of the image.

FIG. 6 is a block diagram illustrating an exemplary computer system 600in which embodiments of the present invention may be implemented. Thisexample illustrates a computer system 600 such as may be used, in whole,in part, or with various modifications, to provide the functions of eyetracking device 120, graphics processing device 130, the game console,the processor/computer 140, and/or other components of the inventionsuch as those discussed above. For example, various functions of eyetracking device 120 and associated processors may be controlled by thecomputer system 600, including, merely by way of example, tracking auser's gaze point, determining an anticipated gaze point, controllinggraphics processing device 130, etc.

The computer system 600 is shown comprising hardware elements that maybe electrically coupled via a bus 690. The hardware elements may includeone or more central processing units 610, one or more input devices 620(e.g., a mouse, a keyboard, etc.), and one or more output devices 630(e.g., a display device, a printer, etc.). The computer system 600 mayalso include one or more storage device 640. By way of example, storagedevice(s) 640 may be disk drives, optical storage devices, solid-statestorage device such as a random access memory (“RAM”) and/or a read-onlymemory (“ROM”), which can be programmable, flash-updateable and/or thelike.

The computer system 600 may additionally include a computer-readablestorage media reader 650, a communications system 660 (e.g., a modem, anetwork card (wireless or wired), an infra-red communication device,Bluetooth™ device, cellular communication device, etc.), and workingmemory 680, which may include RAM and ROM devices as described above. Insome embodiments, the computer system 600 may also include a processingacceleration unit 670, which can include a digital signal processor, aspecial-purpose processor and/or the like.

The computer-readable storage media reader 650 can further be connectedto a computer-readable storage medium, together (and, optionally, incombination with storage device(s) 640) comprehensively representingremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containingcomputer-readable information. The communications system 660 may permitdata to be exchanged with a network, system, computer and/or othercomponent described above.

The computer system 600 may also comprise software elements, shown asbeing currently located within a working memory 680, including anoperating system 684 and/or other code 688. It should be appreciatedthat alternate embodiments of a computer system 600 may have numerousvariations from that described above. For example, customized hardwaremight also be used and/or particular elements might be implemented inhardware, software (including portable software, such as applets), orboth. Furthermore, connection to other computing devices such as networkinput/output and data acquisition devices may also occur.

Software of computer system 600 may include code 688 for implementingany or all of the function of the various elements of the architectureas described herein. For example, software, stored on and/or executed bya computer system such as system 600, can provide the functions of eyetracking device 120, graphics processing device 130, the game console,the processor/computer, and/or other components of the invention such asthose discussed above. Methods implementable by software on some ofthese components have been discussed above in more detail.

The invention has now been described in detail for the purposes ofclarity and understanding. However, it will be appreciated that certainchanges and modifications may be practiced within the scope of theappended claims.

1. (canceled)
 2. A system comprising: an eye tracking device; and one ormore processors configured to: determine a gaze direction of a userbased on data from the eye tracking device; determine a gaze point ofthe user on a display device based on an intersection of the gazedirection with the display device; modify an image on the display devicebased on the gaze point; and wherein modifying the image comprisesaltering an image quality of an area of the image based on a distancefrom the area to the gaze point.
 3. The system of claim 2, whereinaltering the image quality of the area of the image based on thedistance from the area to the gaze point comprises: decreasing the imagequality linearly as the distance from the area to the gaze pointincreases.
 4. The system of claim 2, wherein altering the image qualityof the area of the image based on the distance from the area to the gazepoint comprises: decreasing the image quality in a non-linear fashion asthe distance from the area to the gaze point increases.
 5. The system ofclaim 2, wherein altering the image quality of an area of the imagecomprises: increasing the image quality of the area relative to an imagequality of a remainder of the image when the area comprises the gazepoint.
 6. The system of claim 2, wherein altering the image quality ofthe area of the image based on the distance from the area to the gazepoint comprises: decreasing the image quality in a plurality of discretesteps as the distance from the area to the gaze point increases.
 7. Thesystem of claim 6, wherein modifying the image based on the gaze pointcomprises: increasing the image quality at the gaze point.
 8. The systemof claim 6, wherein the plurality of discrete steps comprises: eachdiscrete step comprising an area of the display device defining a shapecentered around the gaze point.
 9. A method comprising: displaying afirst image on a display device; receiving data from a first eyetracking device relating to a gaze direction of a first user;determining a gaze point of the first user on the display device basedon an intersection of the gaze direction of the first user with thedisplay device; modifying an image quality of the first image based on afirst plurality of distances from the gaze point of the first user to aplurality of locations on the first image; and displaying a second imagecomprising the first image with the image quality modified.
 10. Themethod of claim 9, further comprising: receiving data from a second eyetracking device relating to a gaze direction of a second user;determining a gaze point of a second user on the display device based onthe intersection of the gaze direction of the second user with thedisplay device; and wherein modifying the image quality of the firstimage is further based on a second plurality of distances from the gazepoint of the second user to the plurality of locations.
 11. The methodof claim 9, wherein modifying the image quality of the first image isfurther based on: a distance from the first user to the display device.12. The method of claim 9, wherein modifying the image quality of thefirst image comprises: selecting a first area of the first imagecomprising the gaze point of the first user; and changing an imagequality of the first area from a higher image quality proximate to thegaze point of the first user to a lower image quality at a boundary ofthe first area.
 13. The method of claim 12, the lower image quality atthe boundary of the first area of the first image comprises: an imagequality of a second area remaining unchanged, wherein the second areadoes not overlap the first area.
 14. The method of claim 12, whereinchanging the image quality of the first area comprises: decreasing theimage quality continuously from the gaze point to the boundary of thefirst area.
 15. The method of claim 12, wherein the first area comprisesa shape selected from a list consisting of: a circular shape; an ovalshape; a square shape; a rectangular shape; and a polygonal shape.
 16. Anon-transitory computer readable medium having instructions storedthereon executable by a computing device to cause the computing deviceto perform operations comprising: displaying an image on a displaydevice; receiving data from an eye tracking device relating to a gazedirection of a user; determining a gaze point of the user on the imagebased on an intersection of the gaze direction with the display device;and modifying an image quality of the image based on a plurality ofdistances from the gaze point to a plurality of locations on the image.17. The non-transitory computer readable medium of claim 16, whereinmodifying the image quality of the image is further based on: the eyetracking device detecting a change in the gaze direction.
 18. Thenon-transitory computer readable medium of claim 17, wherein modifyingthe image quality comprises: increasing the image quality of the imagewhen the change in the gaze direction is detected.
 19. Thenon-transitory computer readable medium of claim 17, wherein modifyingthe image quality comprises: decreasing the image quality of the imagewhen the change in the gaze direction is detected.
 20. Thenon-transitory computer readable medium of claim 16, wherein modifyingthe image quality comprises: decreasing the image quality linearly asthe plurality of distances from the locations on the image to the gazepoint increases.
 21. The non-transitory computer readable medium ofclaim 16, wherein modifying the image quality comprises: decreasing theimage quality in a plurality of discrete steps extending radially fromthe gaze point.